Skip to content

fix(e2e): teardown patience matches prod cascade duration (~30–90s) - #2201

Merged
hongmingwang-moleculeai merged 1 commit into
stagingfrom
fix/e2e-teardown-patience
Apr 28, 2026
Merged

fix(e2e): teardown patience matches prod cascade duration (~30–90s)#2201
hongmingwang-moleculeai merged 1 commit into
stagingfrom
fix/e2e-teardown-patience

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Summary

`E2E Staging SaaS` has been failing every cron + push run since 2026-04-27 with:

```
⚠️ LEAK: org e2e-… still present post-teardown (count=1)
exit code 4
```

Root cause: the teardown gave the cascade 30s (curl timeout) + 10s sleep, but the real prod cascade takes 30–90s — EC2 termination alone waits 30–60s on AWS. Observed example: `hongmingwang` DELETE 2026-04-27 took 57s to complete with HTTP 204. The test's 30s timeout aborted the curl mid-cascade and the 10s post-sleep check found `org_instances.status` not yet 'purged'.

Fix

Before After
DELETE curl timeout 30s 120s (--max-time per-call override)
Leak check after single 10s sleep poll up to 60s, 5s steps, exit on first 'purged'

Two cases now handled:

  • Synchronous cascade legitimately taking >30s but <120s → curl returns 200, immediate leak-clean.
  • Curl times out / DELETE 5xx mid-cascade but cascade finishes anyway → poll catches the eventual purge.

The 5-15s estimate in `purge.go`'s comment is the API-call cost only, not the AWS-side time-to-termination it waits on. Async purge would let us drop these limits back to ~15s — future work, called out in code.

Test plan

  • `bash -n` syntax check
  • After merge, watch the next cron e2e-staging-saas run go green (next scheduled fire ~09:23 UTC)
  • OR `gh workflow run e2e-staging-saas.yml` to force an immediate run

Unblocks

Once green, this is the prereq for wiring auto-promote on green staging E2E (the doc-aligned alternative to standing up a canary fleet at <20 paying tenants). Tracked separately.

🤖 Generated with Claude Code

E2E Staging SaaS has been failing on every cron + push run since
2026-04-27 with `LEAK: org … still present post-teardown (count=1)`,
exit 4. Root cause: the curl timeout on the teardown DELETE was 30s
and the post-DELETE leak check was a single 10s sleep — but the
DELETE handler runs the full GDPR Art. 17 cascade synchronously,
including EC2 termination which AWS reports in 30–60s. Real-world
wall time on a prod-shaped run was 57s on 2026-04-27 (hongmingwang
DELETE); the 30s curl timeout aborted the request mid-cascade and
the 10s post-sleep check found the row still present (status not
yet 'purged').

Two-part fix to match real cascade timing:

1. DELETE curl gets its own --max-time 120 (was 30) so the
   synchronous cascade has room to complete in-band.
2. The leak check polls up to 60s for status='purged' instead of
   one rigid 10s sleep. Covers two cases:
   - DELETE returns 5xx mid-cascade but the cascade finishes anyway
     (we still observe a clean state).
   - DELETE legitimately exceeds 120s — eventual-consistency catches
     the eventual purge instead of false-flagging a leak.

The 5–15s estimate in `molecule-controlplane/internal/handlers/
purge.go`'s comment is the API-call cost only, not the AWS-side
time-to-termination it waits on. The async-purge refactor noted in
that comment would let us drop these timeouts back to ~15s — file
that under future work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged via the queue into staging with commit 5c19c53 Apr 28, 2026
16 checks passed
HongmingWang-Rabbit added a commit that referenced this pull request Apr 28, 2026
staging → main: e2e teardown patience (#2201) one-time bridge
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…empfile (resolves #2198, #2199)' (#2201) from fix/e2e-chat-readiness-curl-tempfile-2198 into main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant